DOCUMENT RESUME 



ED 473 102 


HE 035 641 


TITLE 


The Factor Structure Underlying Perceived College Outcomes . 
AIR 2002 Forum Paper. 


PUB DATE 


2002-06-00 


NOTE 


24p.; Paper presented at the Annual Forum for the Association 
for Institutional Research (42nd, Toronto, Ontario, Canada, 
June 2-5, 2002) . 


PUB TYPE 
EDRS PRICE 


Reports - Research (143) — Speeches/Meeting Papers (150) 
EDRS Price MFOl/PCOl Plus Postage. 


DESCRIPTORS 


^Achievement Gains; ^College Students; Factor Analysis; 

* Factor Structure; Higher Education; ^Individual Development; 
^Student Attitudes; Student Characteristics 


ABSTRACT 




The purpose of this study was to examine the factor structure 
that underlies students* perceptions of intellectual progress and personal 
growth and to suggest additional ways of using survey information. The study 
was based on a sample of 68,650 students who completed the American College 
Testing College Outcomes between 1999 and 2001. From the Progress scale of 26 



items, researchers identified four factors: General Education and Skills; 
Critical Thinking; Science, Mathematics, and Technology; and Career 
Preparation. From the Growth scale of 36 items, four other factors were 
identified: Academic and Whole Person Skills; Social Interaction Skills, 
Political and Societal Awareness; and Personal and Spiritual Values. These 
two sets of factors were similar across institution type, especially with 
regard to items in each factor that had the highest factor loadings. In 
addition, factor means were computed for each institutional type so that 
colleges can compare their results with those of similar institutions. 
(Contains 12 tables and 22 references.) (Author/SLD) 



o 

ERIC 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



HE 035 641 



O 



The Factor Structure Underlying Perceived College Outcomes 



CO 

r- 



g 



Presented at 



42"'* Annual Association for Institutional Research (AIR) Forum 
Toronto, Canada 

June 3, 2002 



™ reproduce and 
disseminate THIS MATERIAL HAS 
been granted by 






1 



resources 

information center (ERIC) 



U.S. DEPARTMENT OF EDUCATION 
Office of Educalional Research and Improvemenl 
EDUCATIONAL RESOURCES INFORMATION 
y CENTER (ERIC) 

Ei This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI Dosition or oolicv. 



BEST COPY AVAILABLE 







Abstract 



The Factor Structure Underlying Perceived College Outcomes 

The purpose of this study was to examine the factor structure that underlies students' perceptions of 
intellectual progress and personal growth and to suggest additional ways of using survey information. 
The study was based on a sample of 68,650 students who completed the ACT College Outcomes'^ 
between 1999 and 2001. From the Progress scale of 26 items, we identified four factors: General 
Education and Skills-, Critical Thinking-, Science, Mathematics, and Technology, and Career Preparation. 
From the Growth scale of 36 items, we identified four other factors: Academic and Whole Person Skills, 
Social Interaction Skills, Political and Societal Awareness, and Persona! and Spiritual Values. These two 
sets of factors were similar across institution type, especially with regard to items in each factor that had 
the highest factor loadings. In addition, we computed factor means for each institutional type so that 
colleges can compare their results with those of similar institutions. 
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The Factor Structure Underlying Perceived College Outcomes 

Higher education's evaluation and review teams, accreditation agencies, policy makers and governing 
boards have in common interest in assessing college outcomes. Education finance experts want to know 
if expenditures and investments are worthwhile. College administrators sometimes need ready answers 
to questions about how students feel about aspects of their college experience. A systematic way to 
assess the student pulse and report it in a meaningful way is for some, imperative. As one provost from 
a private university put it, "Sometimes I have to produce quick answers to questions about how our 
students are doing, and when I need answers, I need them now" (W. L. Lester, personal 
communication, June 1999). 

Changes in technology and in world affairs require new perspectives and directions for colleges to 
consider as they monitor student growth and progress. Students now want to know about aspects of the 
institution that relate to quality; in this era of the internet and other technologies they are able to learn a 
great deal about the quality of education at an institution. Students have the option of searching for 
institutions that appear to offer quality instruction and cost-effective education. 

Administrators and policy makers also need to know about quality and cost-effective education. They 
have urgent and often unexpected needs to know about programs in order to allocate and reallocate 
existing resources, provide evidence of the need to continue or discontinue support for given areas, and 
plan for maximum effectiveness in all programs. Ewell (1985) states that student outcome results can 
"be used to improve retention and recruitment strategies, to identify problems within particular programs 
or curricula, or to establish the need for increasing the emphasis on particular skills areas across the 
curriculum" (p. 2). By providing insight into students' perceptions of their college experience, feedback 
from surveys help institutions address planning needs, inform policy makers and curriculum planners, 
triangulate in depth to improve student programs and services, and provide regional and professional 
accrediting bodies with documentation of program effectiveness. 

The survey process may also benefit students. Students make progress intellectually and experience 
personal growth as college years pass, but some may be more aware of their growth than others. Some 
students may have higher levels of consciousness of their progress than others. Their experiences differ, 
their knowledge bases differ, and their motivations differ. Using surveys to assess student perceptions of 
progress and growth may help students develop greater awareness of how college can assist their 
development— their progress and growth. This form of self-assessment may prompt students to become 
more aware of their progress and growth during college. 
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When national user norms are available, an institution can compare its survey results to results from 
comparable institutions. One survey instrument that coileges and universities use to collect students' 
perceptions of their college experience is the ACT College Outcomes Survey {COS) (ACT, 1988). ACT 
designed the COS to produce information at the item ievel, permitting schools to use the data in a variety 
of ways. Factor analysis conveniently reduces the items to a smaller number of concepts or constructs 
with a common underlying theme. When factors perform well, institutions wili have the option of using 
factor means to compare groups of students. 

The purpose of this study was to examine the factor structure that underlies students' perceptions of 
intellectual progress and personal growth and to suggest additional ways of using survey information. 
Items from several sections of the COS, when factor analyzed to identify underlying student responses, 
can enable users to summarize and interpret group differences relative to a limited set of constructs 
rather than to many individual items. We factor analyzed college outcomes items in two areas— 
perceptions of progress with cognitive skills and perceptions of personal growth. 

Each of the two COS sections selected for this study contain items that provide stand-alone measures of 
college outcomes and are used as such by colleges and universities throughout the country. With items 
identified with factors by institution type (public & private, 2-year and 4-year, large and small, and the 
network of colleges), users can compare their own students' responses to those of comparable 
institutions and do so by using factor means. 



Methods 

Data for the Study 

Existing data for this study were retrieved from the most recent national user norms files for colleges and 
universities that had administered the ACT COS during the three-year period prior to January 2002. The 
sample for the national user norms— and for this study — consisted of 68,650 (of approximately 75,000) 
student records. For the norms, this number was arrived at after some records were eliminated to 
guarantee that no individual, institution, or system was over-represented. This sample included technical 
and 2-year colleges (33%), bachelor's degree granting institutions (17%), master's degree granting 
institutions (32%), and institutions granting doctorate or equivalent degrees (19%). Of the 68,650 
records in the sample, 73% were from public and 27% were from private colleges; 67% were from 4- 
year and 33% were from 2-year colleges; and 56% were from small and 44% were from large colleges. 
About 10% of the 68,650 records were from a network of 23 colleges. In all, the sample included 
students from 89 public and 41 private colleges located in 38 US states and consisted of 63% females 
and 37% males. All but about 7% of the total group aspired to a lifetime goal of a bachelor's degree or 
higher— bachelor's (25%), master's (47%), and doctorate or equivalent (21%). 
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Instrument 

The COS was designed to collect student perceptions of aspects of the college experience and the 
institution after the student has been exposed to the college environment for a reasonable period of time. 
The outcome statements on the instrument are broad enough in scope to be applicable to most 
postsecondary institutions, yet specific enough to provide data that can be translated into institutional 
action. Section II of the COS includes two dual-scaled sub-sections, part A and part D, each of which has 
spaces at the left and at the right to collect student responses. We limited this study to data from only 
one of the two scales in each section, the one to the right of the 26 items in Section IIA and the one to 
the left of the 36 items in Section IID. When we refer to the Progress study , we are referring to results 
from the scale on the right side of Section IIA that focuses on progress made at this college in cognitive 
skills and intellectual. growth. When we refer to the Growth study, we are referring to the scale on the 
left side of Section IID that focuses on personal growth since entering this college. For background 
information on students, we used items from Section I, Background Information, and from a form 
submitted by institutions when they mail the surveys to ACT for scoring, analysis, and reporting. 

Procedures and Definitions 

We used frequencies and percentages to describe colleges and students in the sample. We performed 
factor analyses on these two sections of the survey to observe underlying factor structures. For the 
factor analysis, we used a principal components method followed by varimax rotation to identify factors 
for each of the two sets of items. We performed the same type of analysis for the total group and for 
each of seven subgroups. In tables for this study's results, we identify institution types by abbreviations 
at the head of columns containing factor loadings for items listed at the left. Specifically, TOT refers to 
the total group, PUB to public colleges, PVT to private colleges, 4Y to 4-year colleges, 2Y to technical and 
2-year colleges, LG to large colleges and SM to small colleges, and NET for the network of colleges. 

Items assessing perceptions of both Progress and Growth sections are based on 5-point one-dimensional 
scales, where the five options in both sections are very much, much, moderate (average), little, and 
none. In the analyses of each of the two sections, we assigned the following values: S= very much, 
^=much, "immoderate (average), 2=little, and l=none. Although the Growth section of the survey has a 
sixth option, not a goal of mine, we converted any responses to that option to the value of the previous 
column, none, with the value of 1. In our discussions of these 5-point scales, we use the word moderate 
to refer to moderate (average). 

Number of records analyzed for the Progress section. In preparation for analysis of each section, we 
eliminated records with inadequate numbers of valid responses in the section. In the analysis of the 26 
items— the Progress serX\on, we eliminated records in which more than 17 of the 26 possible responses in 
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that section were missing (i.e., if fewer than 9 items had valid responses). We eliminated records in which 
responses to all of the 13 items in the second column were missing. Altogether for analysis of the Progress 
section, we retained a usable total of 66,128 of the 68,650 records, a decrease of 2,522 records. 

Number of records analyzed for the Growth section. In the analysis of the 36 items— the Growth section, 
we eliminated records that contained responses of not a goal of mine^ox more than 30 of the 36 items. For 
remaining records containing any not a goal of mine responses, we assigned to those responses the value 
of a none response. We eliminated records that contained more than 20 missing. We eliminated records 
with more than 17 of the 18 responses in the second column missing. Finally, we retained only those 
records that had valid responses to at least 12 of the 36 items. Using these screening procedures, we 
retained 64,061 of the 68,650 records for analysis of the Growth section, a decrease of 4,589 records. 

Means substitution. For both sections, we substituted item means for the remaining missing values because 
full-rank matrices were required for the analyses. We considered multiple imputation procedures, but 
because less than 1% of the data was missing, we decided to use simple means substitution. Given the 
number of missing values, this procedure likely had little impact on the results of the study. 

Factor analyses. To explore both sections of outcomes items in the COS and to do so for each subgroup, 
we ran separate factor analyses— one for the 26 items in the Progress sedi\on and one for the 36 items in 
the G/ohT/ 7 section. In the Results section factors for the Progress appear in Tables 1 through 4; 
factors for the Growth study appear in Tables 5-8. 

Factor means. The factor means were calculated as averages of the means of items with factor loadings at 
or above the factor loading cutoff point of .50. A summary of factor means for the total group and for each 
institution type appear in Table 9. Factor means and standard deviations for the total group and for public 
and private college groups appear in Tables 10, 11, and 12. 

We used a minimum eigenvalue of 1 (MINEIGEN=1) and a factor loading of .50 in order for them to be 
assigned to factors. Furthermore, with few exceptions, items in a given section were assigned almost 
exclusively to no more than one factor. For convenience in preparing the tables, we multiplied the factor 
loadings by 100 to eliminate the decimal point. 



Results 

Each of the initial principal component analyses yielded four factors. For the total group of respondents in 
the Progress sX.\i6\/ , the first of these unrotated factors had an eigenvalue of 11.51 and accounted for 
44.3% of the variance in the model. Similarly, for the total group of respondents for the G/r?nT/? study, the 
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first of these unrotated factors had an eigenvalue of 17.05 and accounted for 47.4% of the variance in the 
model. 

After the orthogonal (varimax) rotation of the initial principal components (factors), the eigenvalues for the 
four factors identified in the Progress sec\i\or\ were 5.19, 4.47, 3.31, and 2.80 and accounted for 60.6% of 
the variance; the eigenvalues for the four factors identified in the GroM/f/? section were 7.73, 4.98, 4.92, and 
4.09 and accounted for 60.3% of the variance. All but one of the 26 Progress items. Item 10, Further 
developing my study skills, had a loading of at least .50 on one or more of the factors. Ail but four of the 
36 Growth items had a loading of at least .50 on one or more of the factors. These four items were Item 7, 
Actively participating in volunteer work to support worthwhile causes-, Item 21, Learning how to manage 
finances (personal, family, or business)-. Item 23, Developing moral principles to guide my actions and 
decisions', and Item 35, Becoming a more effective member in a multicultural society. 

Factor names. We developed descriptions for each of the factors we observed based on the variables that 
loaded most highly on the factor. The names we have applied to each factor may be debated as to their 
accuracy in describing the factor content, but we have examined the content and tried to select useful 
descriptions. The names are intended merely as handles by which we can refer to groups of items that 
appear to represent a single concept (or construct). The names of factors in the Progress are 

General Education and Skills, Critical Thinking, Science, Mathematics, and Technology, and Career 
Preparation. The names of factors in the Growth section are Academic and Whole Person Development, 
Social Interaction Skills, Political & Societal Awareness, and Persona! and Spiritual Values. 

Items with factor loadings of .50 or higher on at least one factor for at least one type of institution are listed 
by institution type in tables 1 through 8. Loadings of .50 or higher by institution type are listed first for 
each factor. For the total group, both the items and their loadings that are not included in the factor appear 
shaded. For institution types, loadings that are not included for that institution type appear shaded. In 
Tables 1, 2, 5, and 8, some items fell below the .50 cutoff for the total group and for some, but not all, 
subgroups. For example, in Table 1, the total group's General Education and Skills iadior consisted of the 
first 12 items, the last of which was Item 5, Thinking objectively about beliefs, attitudes, and values, with a 
factor loading of .51. However, public colleges included the same 12 items in their General Education and 
5Ar///5 factor, plus Item 10, Further developing my study skills. Private colleges, on the other hand, included 
only four items in their General Education and 5M5 factor— four to which all other groups assigned their 
highest loadings in Genera! Education and Skills. To conclude the example, the unshaded factor loadings 
below a subgroup's column header identify items included in that subgroup's Genera! Education and Skills 
factor. This same procedure of identifying items that belong in a given subgroup's factor applies to each of 
the first eight tables. 
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In general, items with the highest loadings on a given factor were the same across all institution types, 
even though the size of the loadings varied modestly by type. Differences by institution type emerged only 
for items with low factor loadings, relative to the loadings of other items for that factor. While factor 
structures by institution type were not identical, there were few differences, the major one being that for 
private colleges with respect to the factors. General Education and Skills Critical Thinking. 

Four Factors from the Section 

Genera! Education and Skills. As shown in Table 1, the Genera! Education and Skills from the analysis 

of the 26 items in the Progress sedi\or\ points to an appreciation of cultural phenomena and the 
development of life skills that may enhance the enjoyment and the benefits of such things. For the total 
group, the two items with the highest loadings on the Genera! Education and S’/r/Z&factor were Item 16, 
Appreciating the fine arts, music, literature, and the humanities {SS) and Item 18, Discovering productive 
and rewarding uses of my talents and leisure time (67). Each subgroup of institutions also had their highest 
Genera! Education and Skills ladiox loadings on these same two items. 

Critical Thinking. Several items that loaded on the Genera! Education and 5^77/5 factor for public colleges did 
not load on that factor for private colleges. Instead, for private colleges, the skills items loaded on their 
Critical Thinking fadiox. This is the major difference in the factor patterns for the entire study. 
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Table 1 



Section IIA— Progress with Cognitive Skills 
Factor 1 — General Education and Skills 



COS Section IIA - Items by Number 


Unstandardized Factor Loadings by 
Type of Institution 


TOT 


PUB 


PVT 


4Y 


2Y 


LG 


SM 


NET 


16. Appreciating the fine arts, music, literature, 
and the humanities 


68 


66 


69 


67 


67 


67 


68 


68 


18. Discovering productive and rewarding uses of 
my talents and leisure time 


67 


66 


68 


68 


65 


68 


67 


67 


8. Reading with greater speed &. better 
comprehension 


62 


64 


,47 


60 


65 


61 


62 


63 


17. Broadening my intellectual interests 


62 


62 


57 


61 


59 


61 


61 


64 


19. Learning principles for improving physical &. 
mental health 


60 


60 


57 


62 


56 


61 


60 


58 


13. Developing openness to new ideas and 
practices 


56 


58 


45 


55 


50 


53 


54 


56 


12. Learning to formulate and re-shape my 
lifetime goals 


56 


58 


46 


55 


52 


55 


54 


54 


6. Developing my creativity, generating original 
ideas and products 


55 


56 


■.48,. 


53 


54 


52 


55 


58 


7. Improving my writing skill 


54 


57 


40 


50 


61 


50 


57 


56 


9. Speaking more effectively 


54 


57 


39 


50 


60 


50 


56 


54 


11. Listening to and understanding what others 
say 


53 


56 


39 


52 


50 


51 


51 


54 


5. Thinking objectively about beliefs, attitudes, 
&. values 


51 


54 


36 




,47 




49 ; 


53 


10.' Further'developing' my. study skills ; J ’ 


48 


50 


33 


46 


50 


■ 48~"' 


■ 47 


48 


23. Learnihg principles for conserving and 
improving the global environment , ’ 


49 


48 ; 


47 


49 


52 


50 


50 


49 . 



Note . Values are multiplied by 100 and rounded to nearest integer. Values in shaded areas 
factor loading cutoff of 50. 
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Table 2 

Section IIA— Progress with Cognitive Skills 
Factor 2— Critical Thinking 



COS Section IIA - Items by Number 


Unstandarc 

TV] 


lized Factor Loadings by 
pe of Institution 


TOT 


PUB 


PVT 


4Y 


2Y 


LG 


SM 


NET 


3. Learning to think and reason 


76 


75 


77 


76 


77 


76 


77 


76 


2. Developing my problem-solving skills 


75 


75 


73 


75 


76 


75 


75 


75 


1. Drawing conclusions after weighing evidence, 
facts, and ideas 


74 


74 


73 


74 


74 


74 


75 


73 


4. Locating, screening, and organizing 
information 


64 


63 


68 


65 


66 


65 


65 


60 


5. Thinking objectively about beliefs, attitudes, 
and values 


55 


52 


65 


56 


58 


57 


56 


51 


11. Listening to and understanding what others 
say 


52 


; 49,' 


63 


54 


55 


53 


54 


49,: 


13. Developing openness to new ideas and 

• ./ "practices'^T/- ' 


,47 


44. 


58 


48. 


51 


49 


■ 49 . 


44. ■ 


■ 9. Speaking rhore effectively 


40' 


36 


57 


44-: 


" 38 


■■42. 


42 .i 


■30';. 


7. Improving! my writing skill 


40 


36 . 


55 


45 . 


'35 


43: ^ 


41 


:27: 


10. Further 'developing my study skills' 


''42' 


38 


55 


■'.43 


.45 . 


. 41-- ■ 


■■:45 


34;^' 


6. Developing my creativity, generating original 
\ ideasand'products / 


44 


42 


55 


.47 


■46 . 


46 


■ 46 


38., 


;8r Reading with greater speed and better 
. ’ comprehension 


36 


.32 


53 


39 


36 


’. 37 . 


■:f38': ■; 


: 27: 


12.- Learning to fotrnulate and re-rshape my 
. lifetime goals . ' - 


37 


34 


50 


39 


43 . ■ 


38 


■41 


,.32:C^ 



Note . Values are multiplied by 100 and rounded to nearest integer. Values in shaded areas fel 
factor loading cutoff of 50. 



below the 



Science, Mathematics, and Technology. The first four of the five items that loaded on the Science, 
Mathematics, and Technology factor were the same for all institution types (see Table 3). The fifth item. 
Item 24, Effectively using technology (e.g., computers, high tech equipment), had loadings above .50 for all 
but 2-year colleges and the network of colleges, whose loadings were .43 and .48, respectively. The 
loadings for the first two items listed for this factor in Table 3 ranged from a low of .73 for public colleges to 
highs of .88 for 2-year colleges, .89 for small colleges, and .90 for 4-year colleges, large colleges, and the 
network of colleges. 
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Table 3 

Section IIA— Progress with Cognitive Skills 



COS Section IIA - Items by Number 


Unstandarc 

Ty 


lized Factor Loadings by 
pe of Institution 


TOT 


PUB 


PVT 


4Y 


2Y 


LG 


SM 


NET 


25. Learning about the role of science and 
technology in society 


78 


77 


79 


90 


88 


90 


89 


90 


26. Understanding and applying math concepts 
and statistical reasoning 


74 


73 


76 


90 


88 


90. 


89 


90 


22. Drawing conclusions after weighing evidence, 
facts, and. ideas 


71 


70 


74 


67 


64 


66 


66 


65 


23. Locating, screening, and organizing 
information 


63 


63 


64 


60 


58 


58 


60 


55 


24. Effectively using technology (e.g., computers, 
high-tech eguipment) 


53 


52 


55 


57 




54 


51 


■,;48::- 



Note . Values are multiplied by 100 and rounded to nearest integer. Values in shaded areas fel 
factor loading cutoff of 50. 



below the 



Career Preparation. For all institution types, the same four items loaded on the Career Preparation . 
For private colleges, factor loadings were higher for the first two items, in the factor (.76 and .72) than for 
the other two items (.57 and .54). Otherwise, loadings for all institution types were in the upper 60s or 
above. 



Table 4 

Section IIA— Progress with Cognitive Skills 
Factor 4 — Career Preparation 



COS Section IIA - Items by Number 


Unstandarc 

Tyj 


jized Factor L 
pe of Institut 


.oadings by 
ion 


TOT 


PUB 


PVT 


4Y 


2Y 


LG 


SM 


NET 


14. Acquiring knowledge and skills needed for a 
career 


74 


73 


76 


73 


74 


73 


73 


71 


15. Becoming competent in my major 


68 


67 


72 


66 


73 


66 


68 


69 


21. Learning about career options 


68 


69 


57 


68 


69 


70 


69 


72 


20. Developing effective job-seeking skills (e.g., 
interviewing, resume construction) 


65 


66 


54 


66 


66 


67 


66 


72 



Note . Values are multiplied by 100 and rounded to nearest integer. 



Four Factors from the Section 

Academic and Whole Person Skiiis. As shown in Table 5, the Academic and Whoie Person 5/r/7/5 factor for 
the total group consisted of 15 items with factor loadings ranging from .50 to .72. All types of institutions 
had loadings higher than .70 on Item 25, Becoming academically competent All institution types had 
loadings from .63 to .72 for the first seven items in the factor; the first 13 items had loadings above the 
cutoff of .50. Private colleges, 2-year and 4-year colleges, and small colleges had item loadings above .50 
on Item 23, Developing morai principles to guide my actions and decisions, all types except large colleges 
and the network had loadings at .50 or higher on Item 19, Clarifying my personal values. 
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Table 5 

Section IID— Perceptions of Growth 
Factor 1— Academic and Whole Person Skills 



cos Section IID - Items by Number 


Unstandarc 


lized Factor Loadings by 
36 of Institution 


TOT 


PUB 


PVT 


4Y 


2Y 


LG 


SM 


NET 


25. Becoming academically competent 


72 


72 


72 


71 


72 


71 


72 


67 


27. Increasing my intellectual curiosity 


70 


71 


69 


69 


72 


70 


71 


67 


28. Setting long-term or "life" goals 


69 


68 


70 


68 


70 


66 


70 


68 


31. Developing self-confidence 


68 


68 


69 


68 


70 


65 


70 


67 


30. Understanding myself, my talents, and my 
interests 


68 


67 


69 


68 


67 


66 


68 


64 


32. Becoming more willing to change and learn 
new things 


66 


66 


67 


65 


68 


63 


68 


64 


36. Acguiring a well-rounded General Education 


66 


66 


67 


64 


70 


64 


68 


63 


34. Improving my ability to stay with projects 
until they are finished 


63 


61 


65 


63 


62 


60 


65 

1 


58 


29. Constructively expressing both emotions and 
ideas 


61 


59 


64 


61 


61 


58 


63 


59 


26. Developing productive work relationships 
with both men and women 


61 


59 


63 


62 


59 


57 


62 


56 


20. Developing a sense of purpose, value, and 
meaning for my life 


59 


58 


61 


60 


59 


57 


60 


58 


24. Acquiring appropriate social skills for use in 
various situations 


59 


57 


62 


60 


58 


55 


61 


55 


22. Dealing fairly with a wide range of people 


54 


52 


58 


55 


53 


50 


56 


51 


23. Developing moral principles to guide my 
actions and decisions 


50 


48 


54 


51 


50 


46^: 


52 


47 


19. Clarifying my personal values 


51 


50 


53 


52 


50 


49 


52 


'48 ; 


21. Learning how to manage finances (personal, 
family, dr business) , ; \ 


46 


•43 . 


51 


48 


43 


, 43. 


48 : 


41 



Note . Values are multiplied by 100 and rounded to nearest integer, 
factor loading cutoff of 50. 



Values in shaded areas fe 



I below the 



Social Interaction Skills. Table 6 shows the seven items with loadings above the cutoff of .50 for all 
institution types. The loadings were very nearly the same by institution type. In order of the size of the 
loading, the seven items for this factor were: Item 1, Becoming an effective team member, Item 4, 
Improving my ability to relate to others', Item 2, Becoming more willing to consider opposing points of view. 
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Table 6 

Section IID— Perceptions of Growth 
Factor 2— Social Interaction Skills 



COS Section IID - Items by Number 


Unstandarc 

Tvi 


lized Factor Loadings by 
pe of Institution 


TOT 


PUB 


PVT 


4Y 


2Y 


LG 


SM 


NET 


1. Becoming an effective team or group 
member 


70 


71 


69 


69 


72 


70 


70 


72 


4. Improving my ability to relate to others 


68 


68 


69 


68 


69 


69 


68 


66 


2. Becoming more willing to consider opposing 
points of view 


66 


65 


67 


65 


67 


65 


66 


62 


6. Developing leadership skills 


63 


63 


64 


63 


64 


63 


64 


65 


3. Interacting well with people from cultures 
other than my own 


63 


63 


62 


63 


62 


63 


62 


62 


8. Learning to be adaptable, tolerant, and 
willing to negotiate 


60 


61 


59 


60 


61 


61 


60 


60 


5. Preparing to cope with changes as they occur 
(e.g., in career, relationships, lifestyle) 


59 


59 


60 


58 


61 


57 


60 


58 



Note . Values are multiplied by 100 and rounded to nearest integer. 



Item 6, Developing leadership skills, Item 3, Interacting will with people from cultures other than my owrr, 
Item 8, Learning to be adaptable, tolerant, and willing to negotiate, and Item 5, Preparing to cope with 
changes as they occur (e.g., in career, relationships, lifestyle. 



Political and Societal Awareness. In Table 7, we see a similar degree of unanimity in that with one 
exception on the last item for private colleges, the same seven items for all institution types loaded on the 
Political and Societal Awareness ^<±or. All institution types had factor loadings of .70 or above for the first 
three items— Item 12, Becoming more aware of local and national political and social issues, l\.em 11, 
Preparing myself to participate effectively in the electoral process, and Item 10, Becoming more aware of 
global and international issues/events. 
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Table 7 

Section IID— Perceptions of Personal Growth 



Factor 3— Politica 



& Societal Awareness 



COS Section IID - Items by Number 


Unstandarc 

Tyi 


lized Factor Loadings by 
pe of Institution 


TOT 


PUB 


PVT 


4Y 


2Y 


LG 


SM 


NET 


12. Becoming more aware of local and national 
political and social issues 


81 


81 


82 


81 


82 


79 


82 


80 


11. Preparing myself to participate effectively in 
the electoral process 


77 


76 


78 


76 


79 


74 


79 


75 


10. Becoming more aware of global and 
international issues/events 


72 


72 


73 


71 


74 


70 


74 


71 


13. Gaining insight into human nature through 
the study of literature, history, and the arts 


70 


72 


64 


68 


74 


69 


71 


69 


14. Recognizing my rights, responsibilities, and 
privileges as a citizen 


63 


63 


65 


64 


61 


63 


63 


63 


15. Becoming sensitive to moral injustices and 
ways of avoiding or correcting them 


58 


59 


55 


58 


57 


60 


57 


58 


16. Understanding religious values that differ 
from my own 


53 


56 


::;’-45-v 


50 


57 


54 


53 


52 



Note . Values are multiplied by 100 and rounded to nearest integer. The value in the shaded area fell 
the factor loading cutoff of 50. 



Delow 



Personal and Spiritual Values. Table 8 shows the items and item loadings for the Persona! and Spiritual 
Values factor. The first four items were the same for all institution types and the loadings for each item 
were similar. The next item loaded on the factor for all except private colleges and 4-year colleges. The 
remaining three items were not all included in the factors for public, large, or small colleges or the network 
of colleges: Only for 2-year colleges was the factor loading for Item 21, Learning how to manage finances 
(personal, family, or business), high enough to be included. Unlike other types of institutions, private 
colleges and 4-year colleges had loadings at or above the cutoff (.56 and .50, respectively) for Item 16, 
Understanding religious values that differ from my own. Private colleges also had a loading above the 
cutoff for Item 9, Seeking and conveying the spirit of truth. 
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Table 8 

Section IID— Perceptions of Personal Growth 
Factor 4— Personal and Spiritual Values 



cos Section IID - Items by Number 


Unstandarc 

Tvi 


lized Factor Loadings by 
pe of Institution 


TOT 


PUB 


PVT 


4Y 


2Y 


LG 


SM 


NET 


33. Developing my religious values 


67 


66 


70 


69 


63 


67 


66 


67 


18. Learning how to become a more responsible 
family member 


64 


65 


61 


61 


67 


63 


64 


63 


23. Developing moral principles to guide my 
actions and decisions 


57 


58 


56 


57 


57 


59 


56 


59 


19. Clarifying my personal values 


52 


53 


50 


51 


54 


53 


52 


54 


17. Taking responsibility for my own behavior 


51 


53 


• 49 


48 


56 


51 


53 


53 


16. Understanding religious values that differ 
■ from my own ^ 


'48 


45 


56 


50 


47 ' 


46 


49 


47 


, 9. Seeking and conveying the spirit of truth 


.46 


44 


53 


48 


■ 42 


44 - 


46 


47 ■ 


21. -Learning how to manage finances (personal, 
family, 6r;business) ■ L T ■ 


■43 


49 


26 


.36 


54 


.48 


.41, 


• 45 .. . 



Note . Values are multiplied by 100 and rounded to nearest integer. Values in shaded areas fel 
factor loading cutoff of 50. 



below the 



Factor Means 

Factor means for each of the eight factors are presented in Table 9. Factor means, the number, and 
standard deviations for the total group, for public colleges, and for private colleges are presented in Tables 
10, 11, and 12. Each factor mean represents a simple average of item means included in that factor, i.e., 
those items with factor loadings of .50 or above. The items for each factor tended to be the same across 
institution types, but only two factors contained the same items for all institution Career Preparation 

(4 items) and Soda! Interaction Skiiis{7 items). 



Two additional factors contained the same items for all institution types with two exceptions. First, the item 
with the lowest loading on Science, Mathematics, and Technology excluded from that factor by 2-year 
colleges and the network of colleges (see Table 3). Second, the item with the lowest loading on Political 
and Societal Awareness \N^s excluded from that factor by private colleges (see Table 7). 

In the remaining factors, all the items with the highest loadings on the factor were identical for all 
institution types. Among items with lower loadings, the only major exception to this pattern was that for 
items that other institution types loaded on the General Education and 5A///5 factor, private colleges loaded 
onto \he Critical Thinking 



For the total group, the factor with the highest factor mean in the Progress sedion was Critical Thinking 
(3.79). Comparisons of factor means by institution type can only be done with caution for this factor, 
because private colleges had 13 items that loaded on this factor while public colleges only had five items 
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that loaded on it. (See Table 9 for factor means; see Tables 2, 10, and 12 for descriptions of the Critical 
Thinking Parlor and comparisons of private colleges with other types of colleges.) 

The factor in the Progress si&rX\ox\ with the total group's lowest mean was Science, Mathematics, and 
Technology {3.29). The first four items that loaded on. this factor were the same. The fifth item, Item 24, 
Effectively using technology (e.g., computers, high tech equipment), had item loading cutoffs for 2-year 
colleges and the network just below the .50 cutoff — .43 and .48, respectively (see Table 3). The factor 
means by institution type, although not based on identical items, ranged from lows of 3.22 for 2-year and 
3.23 for large colleges to 3.33 for the network of colleges. (See Table 9 for factor means; see Table 3 for 
items that loaded on the factor.) 

For the total group in the Growth section, the factor with the highest mean was Academic and Whole 
Person Skills {3.S9). The factor for the total group consisted of 15 items with only slight variations by 
institution type in either the size of the loading or in the factor mean. The last three of the 15 items were 
not included in the factor means of two or three of the institution types; a sixteenth item was included in 
the factor mean for only one type — private colleges (see Table 5). By institution type, the factor mean 
ranged from lows of 3.84 for 2-year colleges and 3.83 for the network of colleges to highs of 3.92 for 4- 
year colleges and 3.94 for private colleges. 

In the Growth section for the total group, the lowest factor mean was for Political and Societal Awareness 
(3.34). The seven items included in this mean were identical except for the last one, Understanding 
religious values that differ from my own, which was excluded from the factor by private colleges by the .50 
cutoff (see Table 7). By institution type, the factor means ranged from 3.21 to 3.43. 
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Table 9 



Factor Means by Type of Institution 



Eight Factors 


Unstandardized Factor Loadings by 
Type of Institution 


TOT 


PUB 


PVT 


4Y 


2Y 


LG 


SM 


NET 


From the COS Section IIA -Progress 


















FI — General Education and Skills 


3.54 


3.51 


3.44 


3.53 


3.46 


3.42 


3.51 


3.41 


F2 — Critical Thinking 


3.79 


3.77 


3.74 


3.81 


3.73 


3.78 


3.80 


3.69 


F3 — Science, Mathematics, and 
Technology 


3.29 


3.29 


3.28 


3.28 


3.22 


3.23 


3.33 


3.13 


F4 — Career Preparation 


3.65 


3.63 


3.68 


3.64 


3.67 


3.57 


3.71 


3.55 


• From the COS Section IID — Growth 








- 










FI — Academic and Whole Person Skills 


3.89 


3.88 


3.94 


3.92 


3.84 


3.89 


3.90 


3.83 


F2 — Social Interaction Skills 


3.83 


3.80 


3.91 


3.87 


3.76 


3.82 


3.84 


3.75 


F3 — Political & Societal Awareness 


3.34 


3.29 


3.43 


3.38 


3.24 


3.30 


3.37 


3.21 


F4 — Personal and Spiritual Values 


3.74 


3.70 


3.75 


3.63 


3.68 


3.68 


3.78 


3.64 



Note . The configuration of items that define a factor may vary from group to group. The unshaded portions 
of Tables 1 to 8 indicate which items are included in the definition of each subgroup's factors and its factor 
mean. 



Table 10 

Factor Means For the Total Group of Colleges 





N 


Mean 


Std Dev 


^ A Perceptions of Progress-COS Section IlA 


FI - General Education and Skills 

Items A16, A18, A17, A8, A19, A13, A12, A6, A7, A9, All, A5 


66,128 


3.54 


0.73 


F2 - Critical Thinking 

Items A3, A2, Al, A4, A5, All 


66,128 


3.79 


0.70 


F3 - Science, Mathematics, and Technology 

Items A25, A26, A22, A23, A24 


66,128 


3.29 


0.84 


F4 - Career Preparation 

Items A14, A15, A21, A20 


66,128 


3.65 


0.82 


PerceDtions of Growth--COS Section II D 


FI - Academic and Whole Person Skills 

Items D25, D27, D28, D31, D30, D32, D36, D34, D29, D26, D20, D24, 
D22, D23, D19 


64,061 


3.89 


0.74 


F2 - Social Interaction Skills 

Items Dl, D4, D2, D6, D3, D8, D5 


64,061 


3.83 


0.72 


F3 - Political & Societal Awareness 

Items D12, Dll, DIO, D13, D14, D15, D16 


64,061 


3.34 


0.91 


F4 - Personal and Spiritual Values 

Items D33, D18, D23, D19, D17 


64,061 


3.74 


0.87 



Note . The A or D preceding each item number refers to COS Section IIA or Section IID. 
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Table 11 

Factor Means For Public Colleges 





N 


Mean 


Std Dev 


Perceptions of Progress-COS Section II A 


FI - General Education and Skills 

Items A16, A18, A17, A8, A19, A13, A12, A6, A7, A9, All, AS, AlO 


■48,493 


3.S1 


0.73 


F2 - Critical Thinking 

Items A3, A2, Al, A4, AS 


48,493 


3.77 


0.71 


F3 — Science, Mathematics, and Technology 

Items A2S, A26, A22, A23, A24 


48,493 


3.29 


0.83 


F4 - Career Preparation 

Items A14, AIS, A21, A20 


48,493 


3.63 


0.82 


Perceptions of Growth-COS Section II p 


FI - Academic and Whole Person Skills 

Items D2S, D27, D28, D31, D30, D32, D36, D34, D29, D26, D20, D24, 
D22, D19 


46,730 


3.88 


0.74 


F2 - Social Interaction Skills 

Items Dl, D4, D2, D6, D3, D8, DS 


46,730 


3.80 


0.73 


F3 - Political & Societal Awareness 

Items D12, Dll, DIO, D13, D14, DIS, D16 


46,730 


3.29 


0.91 


F4 — Personal and Spiritual Values 

Items D33, D18, D23, D19, D17 


46,730 


3.70 


0.88 



Note . The A or D preceding each item number refers to COS Section IIA or Section IID. 



Table 12 

Factor Means For Private Colleges 





N 


Mean 


Std Dev 


Perceptions of Progress-COS Section II A 


FI - General Education and Skills 

Items A16, A18, A17, A19 


17,635 


3.44 


0.86 


F2 - Critical Thinking 

Items A3, A2, Al, A4, A5, All, A13, A9, A7, AlO, A6, A8, A12 


17,635 


3.74 


0.69 


F3 — Science, Mathematics, and Technology 

Items A25, A26, A22, A23, A24 


17,635 


3.28 


0.86 


F4 — Career Preparation 

Items A14, A15, A21, A20 


17,635 


3.68 


0.81 


“T Perceptions of Growth— COS Section II D 


FI — Academic and Whole Person Skills 

Items D25, D27, D28, D31, D30, D32, D36, D34, D29, D26, D20, D24, 
D22, D23, D19, D21 


17,331 


3.94 


0.72 


F2 — Social Interaction Skills 

Items Dl, D4, D2, D6, D3, D8, D5 


17,331 


3.91 


0.70 


F3 - Political & Societal Awareness 

Items D12, Dll, DIO, D13, D14, D15 


17,331 


3.43 


0.91 


F4 — Personal and Spiritual Values 

Items D33, D18, D23, D19, D16, D9 


17,331 


3.75 


0.84 



Note . The A or D preceding each item number refers to COS Section IIA or Section IID. 
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Discussion 



Summary of Findings 

Using factor analysis, we identified four factors underlying the COS Section lll\- Progress sca\e (of 26 
\ter(\s)— General Education and Skills, Critical Thinking; Science, Mathematics, and Technology, and 
Career Preparation. We also identified four factors underlying the COS Section IID-GroM/f/? scale of 36 
Academic and Whole Person Skills, Social Interaction Skills, Political and Societal Awareness, and 
Personal and Spiritual Values. In each instance, the factors identified were similar across all institution 
types in as much as items with the highest factor loadings for the total group were also those with the 
highest loadings for each institution type. The major exception occurred among items with lower 
loadings on the General Education and 5M5 factor in that some of the skill-related items associated by 
public college respondents with this factor were instead associated by private college respondents with 
the Critical Thinking factor. Even so, the items with the highest factor loadings for these two factors 
were identical for all institution types, including both public and private college respondents. The only 
other discrepancies of this kind that occurred on factors were among items with loadings near the .50 
cutoff. The consistency of factor loadings across institution types suggests that these factors have broad 
applicability. We calculated factor means, for each institution type. 

Caveats 

The cutoff point of .50 for item loadings was not entirely arbitrary, but as with any cutoff, the argument 
could be made for placing it at some other point. Several items with loadings between .40 and .49 could 
have been considered when labeling the factors we identified. Would adding these items to the factors 
on which they loaded have changed the nature of the factor? Probably not, but a local user might have 
reason to include more of the items than we included in constructing the factors in this study. 

Although the sample is very large, it is a user sample, not a nationally representative sample. Even 
though we made several adjustments to improve it, this does not change the basic fact that it consists of 
student records from institutions that have used the COS. 

Furthermore, as with any factor analysis, the derived factors are constrained by the quality and 
appropriateness of the items that were factor analyzed. While the COS items were developed as a result 
of studies of the literature on college outcomes assessment, the factors identified in this study are limited 
by the content of the items analyzed. 

Given the magnitude of the eigenvalues for the first unrotated factors of both the Progress s&dSon and 
the Growth section (accounting for 44.3% and 47.4% of the variance in the two analyses), one might 
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argue that response set or variations in the overall level of perceived progress/growth outweighed all of 
the findings. Nevertheless, despite the constraints, the rotated factor solutions yielded four factors that 
make sense intuitively and correspond well with findings from outcome studies reported in the literature. 



Uses and Implications of the Factors 

Given that the instrument we have chosen limits us to student perceptions of their college outcomes, are 
these areas appropriate for college administrators and policy makers to be concerned about? Are the 
factors we identified appropriate for colleges to use in assessing college outcomes? In their Carnegie 
Foundation report, Boyer and Levine (1981) used the analogy of a 3-room house to describe areas of 
responsibility for the undergraduate curriculum. Their first room is the academic major, the second is the 
electives area, and the third is general education. The COS instrument focuses more on the latter, even 
if it speaks to some extent to all three. At least one, and arguably several, of the factors identified in this 
study appear to be related to goals of general education. General education is one of the most important 
and most difficult charges of higher education. Many books, articles, and special commissions have been 
inspired by its challenge (Harvard College, 1945; President's Commission on Higher Education, 1947; 
Dressel & Mayhew, 1954; Rudolph, 1977, 1962; Boyer & Levine, 1981; Gaff, 1983, 1989; Weingartner, 
1992). 



General education is usually the responsibility of the entire institution, but too frequently, the 
responsibility is dispersed. The means — usually, the required general education courses— can easily be 
mistaken for the desired ends. With factors that focus on students' perceptions of the desired ends and 
on their actual progress and growth, colleges can learn whether means and ends are appropriately 
related to college outcomes. Although the instrument asks if students feel they have become competent 
in their major and if they have acquired knowledge and skills needed for a career, the responses to these 
questions remain in the realm of perceptions. The instrument asks students about the extent to which 
they have broadened their intellectual interests, increased their intellectual curiosity, and developed 
openness to new ideas, but again, the responses to these questions remain in the realm of perceptions. 
Nevertheless, a large and difficult portion of college outcomes have been addressed— those associated 
with the ends that most colleges hope to achieve. Using the analogy of higher education's 3-room house 
(Boyer & Levine, 1981), these college outcomes include, as the first room, a sense of competence in a 
major area— the faculty's domain; as the second room, a breadth of intellectual and personal interests— 
the student's domain; and as the third room, goals associated with general education— a domain claimed 
by no one but the responsibility of all. 



Based on an examination of literature that analyzes the nature of assessment of college outcomes and 
the rationales for such assessment, we feel the factors identified in this study do indeed point to a 
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sizeable portion of desired college outcomes. The results also appear to match the major areas of 
growth and progress mentioned in the literature on outcomes of higher education. For example, 
Pascarella and Terenzini (1991) reviewed and synthesized 2,600 pieces of research on student 
characteristics likely to be affected by the college experience. A catalog of the college outcomes 
documented in the literature can be summarized by listing topics from the chapter titles of their 900-page 
book. How College Affects Students (1991), that outlines areas that parallel the areas of this study. Their 
book includes such topics as cognitive skills and intellectual growth; verbal, quantitative, and subject 
matter competence; identity, self-concept, and self-esteem; psycho-social changes; attitudes and values; 
moral development; educational attainment; career choice and development; economic benefits; and 
quality of life after college. Indeed, their book was useful in the development of the COS instrument 
because of its degree of thoroughness. Many of its topics appear in items of the progress and growth 
sections of the instrument we used, and many characteristics the authors found to be associated with 
how college affects students are included in the factors identified in the present study. Earlier studies 
have also been done to examine how college affects students. Feldman and Newcomb (1969) looked for 
and found similar college outcomes by examining 1,500 studies conducted over a period of 40 years. 

One caveat that Pascarella and Terinzini (1991) offered in their book was that many of the studies they 
examined were conducted in 4-year colleges. The current study addresses their concern by examining 
perceived outcomes across several institution types, including 2-year colleges. 

Astin (1973) has referred to need for a taxonomy that includes both cognitive and affective dimensions— 
the whole person. He and other members of the Study Group on the Conditions of Excellence in 
American Higher Education issued an influential final report. Involvement In Learning: Realizing the 
Potential of American Higher Education {y\ox^\me.y, et al., 1984), sharing with the education community 
the study group's insights. The report addresses the need for students to do more than acquire 
knowledge; they must acquire other character traits as well. For example, in their first chapter, they 
state, "The United States must become a nation of educated people. Its citizens fnust be knowledgeable, 
creative, and open to ideas. Above all, they should learn how to learn so they can pursue knowledge 
throughout their lives and assist their children in the same quest" (p. 2). On numerous occasions, Astin 
has emphasized that the college experience should make a positive difference to students, not only in 
their knowledge and skills, but also in affective areas such as attitudes (1985, 1984, 1975, & 1973). 

Astin has long been an advocate of student surveys and self-reports (1991). 

In the introduction of Assessing Educational Outcomes, Ewell (1985) stated that there are many different 
typologies of student outcomes, but that all of them make a number of distinctions-three major ones. 
First is the distinction between cognitive and affective outcomes, i.e., we need to distinguish between 
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gains in knowledge and changes in attitudes or values. Second is the distinction between psychological 
and behavioral outcomes, l.e., we need to distinguish between "changes occurring inside a student's 
,head and changes that can be observed directly during and after college" (p. 3). Third is the distinction 
between within-college and after-college outcomes, i.e., we need to distinguish when the outcome 
occurs. In this study, we have tried to address such distinctions. The factors and the items contained in 
each factor can help colleges to make such distinctions as they engage in outcomes assessment. 
Regarding the last distinction, we did not take full advantage of the COS instrument to distinguish 
between contributions of the college versus growth that occurred outside the institution. Nevertheless, 
the sections we did use are preceded by instructions to the student in the Progress s&cWon to indicate 
"progress made at this college toward attainment of that outcome." Instructions in the Growth secWon 
asks students to indicate "the extent of our growth since entering this college (regardless of the extent 
of the contribution made by your experiences at this college." Students are instructed to use the Growth 
section a second time to indicate the extent of the college contribution to their growth. However, we did 
not include responses from the college contribution scale in our study. Our study does, however, address 
the first two distinctions that Ewell described as important in assessments of college outcomes. 

Howard Bowen (1977) reviewed close to 600 studies of college outcomes to see if colleges are worth 
what they cost. He identified outcomes and benefits that accrue to those who attend college- 
intellectual, personal, and economic benefits. Many of the intellectual and personal dimensions he 
identified parallel those of the current study. He concluded that, "Education should be directed toward 
the growth of the whole person through the cultivation not only of the intellect and of practical 
competence but also of the affective dispositions, including the moral, religious, emotional, social, and 
esthetic aspects of the personality. No theme runs more consistently through the goal literature." (p. 

33). K. Patricia Cross (1986) concurred with Bowen's conclusion, adding that this "widely accepted goal 
is what makes the assessment of higher education so difficult" (p. 11). 

The findings of the current study appear to be in line with much of the literature on college outcomes, 
even that of Bowen's conclusion about the "whole person." Factors such as those identified in this study 
can be useful to individual institutions as they move toward defining and assessing appropriate college 
outcomes. 
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